Heads up — smoke-testing this branch's image (ghcr.io/opzkit/database-user-operator:pr-135) on a fresh cluster via make deploy results in CrashLoopBackOff:

flag provided but not defined: -leader-elect
Usage of /manager:
  -health-probe-bind-address string
  -kubeconfig string
  -metrics-bind-address string
  -zap-devel
  ...

config/manager/manager.yaml:33 still passes --leader-elect, but cmd/main.go no longer registers that flag (looks like leader election was dropped or its registration was removed during the refactor). The binary aborts on the unknown flag before the manager starts.

Fixes I can see (in order of preference):

1. Re-add leader-election registration in cmd/main.go — restore the flag + the LeaderElection: enableLeaderElect field on the manager.Options{} literal. This matches the controller-runtime template and keeps the manifest correct.
2. Drop --leader-elect from config/manager/manager.yaml if leader election is intentionally gone for now — but then HA replicas won't coordinate; not what you want.

Local workaround for anyone else hitting this:

kubectl -n database-user-operator-system patch deployment database-user-operator-controller-manager \
  --type=json \
  -p='[{"op":"remove","path":"/spec/template/spec/containers/0/args"}]'

Happy to send a small follow-up PR if you'd like — just confirm whether you want option 1 or 2.

Two more findings from the smoke test on Scaleway managed PG via secretBackend.kubernetes:

1. Default secretName produces an invalid Kubernetes Secret name

With secretBackend.kubernetes: {} and no explicit spec.secretName, the default is rds/<engine>/<databaseName> (e.g. rds/postgres/dbuo_smoketest). The Kubernetes backend then tries to delete/create a Secret by that name and fails:

failed to delete kubernetes secret: invalid resource name "rds/postgres/dbuo_smoketest": [may not contain '/']

The default was AWS-SM-friendly (path-shaped names are fine in Secrets Manager), but it isn't valid for the Kubernetes backend. The backend selection should either:

• override the default name to a DNS-1123-compatible form (e.g. replace / with -), or
• validate at admission and refuse the resource with a clear error rather than failing in the reconcile loop.

2. Stale-state recovery doesn't help when partial reconciles leave orphan users

When the first reconcile creates the PG user but the Secret-write step fails (e.g. connection-string parsing problem on a previous attempt left the user behind, then we fixed the DSN), the next reconcile errors with:

database and/or user exist but secret is missing - cannot recover password
(database exists: false, user exists: true, secret exists: false).
Please delete the Database CR and recreate it, or manually create the secret with the correct password

The suggestion to "delete the Database CR and recreate it" doesn't actually unblock anyone in the kubernetes-backend case until #1 above is fixed, because the finalizer's secret-delete step itself fails on the same /-in-name validation, leaving the CR in a perma-deleting state until the finalizer is patched out.

Workaround for #2 in the meantime: kubectl patch database <name> -p '{"metadata":{"finalizers":[]}}' --type=merge to release the CR, then DROP the orphan PG user manually before recreating with an explicit spec.secretName set to a DNS-1123 string.

Both feel like blockers for the kubernetes backend path. Happy to send a follow-up PR for #1 if you'd like — likely a small change in secrets.NewBackend (or wherever the secretName default is computed) to derive backend-aware defaults.

Third finding: the RBAC marker for Secrets in internal/controller/database_controller.go:55 only grants get;list;watch:

// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch

The Kubernetes backend needs to create, update, and delete Secrets too — currently the controller fails the cleanup path with:

failed to delete kubernetes secret: secrets "dbuo-smoketest-creds" is forbidden:
User "system:serviceaccount:database-user-operator-system:database-user-operator-controller-manager"
cannot delete resource "secrets" in API group "" in the namespace "dbuo-test"

Suggested fix:

// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch;create;update;patch;delete

(Same marker can stay cluster-scoped if secretBackend.kubernetes.namespace may differ from the Database's namespace. If we want to scope it down later, swap to a Role + RoleBinding generated from the same marker via +kubebuilder:rbac:...,namespace=... — but that's a bigger refactor.)

I'll send a small follow-up PR with the marker change unless you'd rather fold it into this one.

Fourth finding (root cause of the catch-22 we kept hitting earlier): on Scaleway managed PostgreSQL, the GRANT step in internal/database/postgres.go:205 fails:

failed to grant user to admin: pq: cannot use special role specifier in GRANT/REVOKE ROLE (XX000)

The offending statement:

grantQuery := fmt.Sprintf("GRANT %s TO CURRENT_USER", quoteIdentifier(username))

CURRENT_USER is a special role specifier in Postgres (alongside CURRENT_ROLE, SESSION_USER, PUBLIC). Some managed Postgres providers — Scaleway managed RDB confirmed, RDS reportedly intermittent depending on parameter group — reject it as a role-recipient and raise XX000 from the role-resolution layer.

Suggested fix: resolve CURRENT_USER to the actual connection role name once at controller startup (or once per connection) and substitute the literal role name into the GRANT:

var adminRole string
if err := c.db.QueryRowContext(ctx, "SELECT current_user").Scan(&adminRole); err != nil {
    return fmt.Errorf("failed to resolve current admin role: %w", err)
}
grantQuery := fmt.Sprintf("GRANT %s TO %s", quoteIdentifier(username), quoteIdentifier(adminRole))

This also gives nicer error messages when GRANT fails, since the literal role name appears in the SQL.

Combined with the earlier findings (catch-22 logic when secret-create fails, /-in-default-secretName for kubernetes backend, missing RBAC verbs, stale --leader-elect flag), the kubernetes backend currently can't complete a single Database reconcile against Scaleway managed PG.

Happy to put together a PR rolling all five fixes if you'd prefer them bundled, or send each separately.

Confirmed the fix locally — replacing CURRENT_USER with the resolved role name lets the GRANT succeed against Scaleway managed RDB. Patch:

--- a/internal/database/postgres.go
+++ b/internal/database/postgres.go
@@ -202,11 +202,14 @@ func (c *PostgresClient) CreateUser(ctx context.Context, username, password str
 			return fmt.Errorf("failed to create user: %w", err)
 		}
 
-		// Grant the new user to the current admin user so we can SET ROLE to it
-		// This is required for CREATE DATABASE ... OWNER to work in managed PostgreSQL (RDS, etc)
-		grantQuery := fmt.Sprintf("GRANT %s TO CURRENT_USER", quoteIdentifier(username))
-		_, err = c.db.ExecContext(ctx, grantQuery)
-		if err != nil {
-			return fmt.Errorf("failed to grant user to admin: %w", err)
+		// Grant the new user to the current admin role so we can SET ROLE to it.
+		// `CURRENT_USER` is rejected as a role-recipient by some managed PG providers
+		// (Scaleway managed RDB raises XX000 "cannot use special role specifier"),
+		// so resolve it to a literal role name first.
+		var adminRole string
+		if err := c.db.QueryRowContext(ctx, "SELECT current_user").Scan(&adminRole); err != nil {
+			return fmt.Errorf("failed to resolve current admin role: %w", err)
+		}
+		grantQuery := fmt.Sprintf("GRANT %s TO %s", quoteIdentifier(username), quoteIdentifier(adminRole))
+		if _, err := c.db.ExecContext(ctx, grantQuery); err != nil {
+			return fmt.Errorf("failed to grant user to admin (%s): %w", adminRole, err)
 		}
 	}

Side-note while testing: on Scaleway managed RDB the _rdb_superadmin trigger auto-grants membership in the new role to the connecting admin during CREATE USER, so the GRANT step ends up a no-op there (the literal-name version succeeds with a NOTICE: ... has already been granted membership ...). The fix doesn't need to be conditional — the GRANT remains correct on RDS-style providers, and idempotent on Scaleway. Just worth knowing for the test matrix.

So in summary, the five blockers found while smoke-testing this branch end-to-end (Scaleway managed RDB + secretBackend.kubernetes):

1. --leader-elect flag in config/manager/manager.yaml no longer registered by the binary.
2. Default secretName (rds/<engine>/<db>) contains /, invalid for the kubernetes backend.
3. RBAC marker for Secrets only grants get;list;watch; needs create;update;patch;delete for the kubernetes backend.
4. Catch-22 reconcile path when userExists && !secretExists is unrecoverable for the kubernetes backend (compounded by 1–3 leaving partial state behind).
5. GRANT %s TO CURRENT_USER rejected by some managed PG providers — needs literal role name.

Fix #5 alone unblocks the happy path (smoke test reaches "user + database + secret created"); the other four make the failure modes much friendlier.

Unrelated suggestion while we're in here: the controller has no version banner on startup, which makes triage painful when running unreleased builds (it took us a few minutes to confirm which exact image we were looking at while debugging the GRANT issue, since pr-135 and a follow-up pr135-current-user-fix rebuild looked identical from the outside).

Standard kubebuilder pattern: stamp version, gitCommit, and buildDate into the binary at link time, then log them once at startup.

// cmd/main.go (top-level vars, populated by -ldflags)
var (
    version   = "dev"
    gitCommit = "none"
    buildDate = "unknown"
)

// inside main(), right before manager start:
setupLog.Info("starting database-user-operator",
    "version", version,
    "gitCommit", gitCommit,
    "buildDate", buildDate,
)

Dockerfile (or the Makefile target that calls go build):

ARG VERSION=dev
ARG GIT_COMMIT=none
ARG BUILD_DATE=unknown
RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build \
    -a -ldflags "-X main.version=${VERSION} -X main.gitCommit=${GIT_COMMIT} -X main.buildDate=${BUILD_DATE}" \
    -o manager cmd/main.go

build-and-scan.yaml already has ${{ steps.meta.outputs.version }} and ${{ github.sha }} available; pass them through as build args. BUILD_DATE can be $(date -u +%Y-%m-%dT%H:%M:%SZ) set in a prior step.

Ergonomic bonus: same vars can back a --version CLI flag and a /version HTTP endpoint on the existing health-probe listener if you want it scrapeable.

Pushed bf1a8f6 resolving the five smoke-test blockers (and one related cleanup):

1. --leader-elect crash — went with the simpler path of hardcoding LeaderElection: true in cmd/manager/main.go and dropping the --leader-elect arg from config/manager/manager.yaml. Documented as breaking for any kustomize users overriding the arg list. Helm values.yaml never passed the flag, so chart users are unaffected. docs/TROUBLESHOOTING.md example updated to drop the flag.

2. Default secretName invalid for kubernetes backend — getSecretNameOrDefault is now backend-aware. AWS / Infisical keep rds/<engine>/<db>; kubernetes uses rds-<engine>-<db> (DNS-1123 compliant). Existing tests still pass since they don't set the kubernetes backend.

3. RBAC verbs — applied your suggested marker verbs=get;list;watch;create;update;patch;delete, regenerated config/rbac/role.yaml.

4. Catch-22 when userExists && !secretExists — replaced the unrecoverable error with a recovery path: rotate password via CreateUser (idempotent — ALTER USER WITH PASSWORD when the user exists), create the database if missing, fall through to the normal secret-write step. Safe because the operator owns the user (stamped by COMMENT ON ROLE at create time).

5. GRANT … TO CURRENT_USER rejected by managed PG — applied your patch verbatim. SELECT current_user resolves the literal role, then GRANT %s TO %s. Confirmed idempotent on Scaleway's _rdb_superadmin-trigger setup.

Version banner via -ldflags is a good suggestion — filing as a separate follow-up rather than folding into this PR to keep the diff focused. Will track once #135 lands.

PR body's "Migration: see Breaking changes" section unchanged; the leader-election arg removal is now also breaking for raw-kustomize users — will add a note to the body.

Sixth blocker (only surfaced once #5's CURRENT_USER patch was in place): on PG 16+ the GRANT defaults changed to NOINHERIT, NOSET, NOADMIN, and CREATE DATABASE x OWNER y / ALTER DATABASE x OWNER TO y now require the executor to be a member of y with the SET privilege. Bare membership is no longer enough.

Reproduces on Scaleway managed RDB (PG 16) — the operator gets to the create-database step and errors:

failed to create database: pq: must be able to SET ROLE "dbuo_smoketest" (42501)

It doesn't reproduce on AWS RDS / Aurora — even on PG 17 — because RDS/Aurora ship an event-trigger on CREATE ROLE that auto-runs something like GRANT new_role TO rds_superuser WITH INHERIT, SET, ADMIN OPTION, papering over PG 16's stricter defaults. Scaleway managed RDB has a similar trigger (_rdb_superadmin shows up in NOTICE: ... has already been granted membership ... by role "_rdb_superadmin"), but it only grants bare membership — no flags. So DBUO works on RDS today, breaks on Scaleway, and would also break on a self-hosted PG 16+ without the same trigger.

Verified the patch:

GRANT dbuo_smoketest TO intersolia_admin WITH INHERIT TRUE, SET TRUE;
-- then
CREATE DATABASE dbuo_smoketest_alt OWNER dbuo_smoketest;  -- succeeds

Suggested fix — combine with #5, since both touch the same GRANT statement:

--- a/internal/database/postgres.go
+++ b/internal/database/postgres.go
@@ -202,11 +202,21 @@ func (c *PostgresClient) CreateUser(ctx context.Context, username, password str
 			return fmt.Errorf("failed to create user: %w", err)
 		}
 
-		// Grant the new user to the current admin user so we can SET ROLE to it
-		// This is required for CREATE DATABASE ... OWNER to work in managed PostgreSQL (RDS, etc)
-		grantQuery := fmt.Sprintf("GRANT %s TO CURRENT_USER", quoteIdentifier(username))
-		_, err = c.db.ExecContext(ctx, grantQuery)
-		if err != nil {
-			return fmt.Errorf("failed to grant user to admin: %w", err)
+		// Grant new user → admin with INHERIT + SET so CREATE/ALTER DATABASE
+		// ... OWNER works on PG 16+ (defaults changed to NOINHERIT, NOSET).
+		// Resolve CURRENT_USER to a literal name first (some managed PG
+		// providers reject CURRENT_USER as a special role specifier — XX000).
+		// Aurora/RDS auto-grant these flags via an event trigger; Scaleway
+		// managed RDB and self-hosted PG 16+ do not.
+		var adminRole string
+		if err := c.db.QueryRowContext(ctx, "SELECT current_user").Scan(&adminRole); err != nil {
+			return fmt.Errorf("failed to resolve current admin role: %w", err)
+		}
+		grantQuery := fmt.Sprintf(
+			"GRANT %s TO %s WITH INHERIT TRUE, SET TRUE",
+			quoteIdentifier(username),
+			quoteIdentifier(adminRole),
+		)
+		if _, err := c.db.ExecContext(ctx, grantQuery); err != nil {
+			return fmt.Errorf("failed to grant user to admin (%s): %w", adminRole, err)
 		}
 	}

PG <16 ignores WITH SET TRUE (the column doesn't exist) but does honour WITH INHERIT TRUE, so the statement works on PG 14/15 too. Verified the syntax is parsed cleanly on PG 16.4.

So the running tally of blockers found smoke-testing this branch on Scaleway managed RDB + secretBackend.kubernetes:

1. Stale --leader-elect flag in config/manager/manager.yaml.
2. Default secretName (rds/<engine>/<db>) contains /, invalid for kubernetes backend.
3. RBAC marker for Secrets only get;list;watch.
4. Catch-22 reconcile when userExists && !secretExists with secretBackend.kubernetes (already addressed in latest pr-135 build with the "rotate password and recreate secret" recovery path — confirmed working).
5. GRANT %s TO CURRENT_USER rejected as special role specifier (XX000).
6. (this comment) PG 16+ requires WITH INHERIT TRUE, SET TRUE on the admin grant.

Plus the unrelated build-info banner suggestion in the previous comment.